Comcast Data Engineer interview questions
Why Data Engineer?
Data engineers design systems that collect, handle, and convert raw data into usable information for data scientists and business analysts to comprehend in a range of scenarios. Their ultimate goal is to make data more available so that businesses may assess and improve their performance.
What is the salary of Data Engineer?
The market for big data specialists has never been higher, and careers as a data scientist or data engineer can be found among LinkedIn's top emerging jobs. As data engineers, many people pursue higher-paying positions. The typical income ranges from $65,000 to $132,000, depending on your experience and extra skills in the area. Wage levels in India range from 3.5 lakhs to 20.2 lakhs, with an average yearly salary of 8.1 lakhs. However, according to these reports, data engineers are the highest-paid professionals in the industry.
Comcast Data Engineer Interview questions and answers
Here are some of the most recent Data Engineer Interview Questions from Comcast. These questions are ideal for new graduates as well as experienced employees. Our PySpark Training in Chennai specialists have answered all of the questions below.
1. In Spark, what is the difference between Dataframe, Dataset, and RDD?
As a result of its tabular nature, a Dataframe has additional metadata that allows Spark to perform specific optimizations on the resulting query. An RDD, on the other hand, is more of a black box of data that cannot be optimized due to the absence of constraints on the operations that may be performed upon it.
However, you can use the RDD method to convert a Dataframe to an RDD, and the DF function to convert an RDD to a Dataframe. Because of the built-in query optimization, it is generally suggested to use a Dataframe wherever possible.
2. coalesce vs repartition () in Spark ()
It prevents a complete shuffling. If the number is known to be reducing, the executor can securely store data on the smallest number of divisions possible, only moving data from the extra nodes to the ones we kept.
3. What is the difference between a map and a flat Map, and when should each be used?
An RDD of length N is transformed into another RDD of length N, and it translates from two lines to two lines-length. Flat Map, on the other hand, takes an RDD of length N and turns it into a collection of N collections, which it then flattens into a single RDD of results.
4. What exactly is the distinction between cache and persist?
The only distinction between cache and persist actions is one of syntax. The cache is a synonym for persist or persist (MEMORY ONLY), implying that cache is just persisting with the default storage level of MEMORY ONLY.
5. CSV file as dataframe in Spark?
Spark-csv is a built-in feature that does not require a separate library. So, as an example, you could
df = spark.
read.
format(“csv”).
option (“header”, “true”).
load(“csvfile.csv”)
In Scala (this works for any format-in delimiter mention “,” for csv, “\t” for tsv etc.)
